The present invention relates generally to systems and methods for monitoring movement of a human eye, and more particularly to systems and methods for real-time monitoring of fatigue and other states of mind in individuals, purposeful communication, and/or controlling devices based upon movement of their eye, eyelid, and/or other components of their eye.
There have been attempts to use movement of the human eye to monitor involuntary conditions, specifically a person""s wakefulness or drowsiness. For example, U.S. Pat. No. 3,863,243 discloses a device that sounds an alarm to warn a person using the device that they are beginning to fall asleep. The device includes a frame similar to a set of eyeglasses onto which is mounted a fiber optic bundle and a photocell that are directed towards the user""s eye when the frame is worn. The fiber optic bundle is coupled to a source of light and a pulse generator to emit light towards the user""s eye.
The photocell detects the intensity of light reflected off of the user""s eye, i.e., either by the eyelid when the eye is closed or the eye surface when the eye is open. Circuitry receives a signal from the photocell, and uses a timer to distinguish between regular blinks, and an extended time period during which the eye is closed, i.e., a time period that may indicate that the person is falling asleep. When a threshold time elapses, an alarm is sounded to notify and/or wake the user. This device, however, requires running wires and fiber optic bundles from the frame to external components, e.g., the pulse generator and the required circuitry, and for this reason, the device may be awkward or inconvenient to use.
Other devices, such as those disclosed in U.S. Pat. Nos. 5,469,143 and 4,359,724, directly engage the eyelid or eyebrow of a user to detect movement of the eye and activate an alarm when a drowsiness condition is detected. These mechanical devices may be mounted directly onto the skin to detect muscle movement or may involve placing a mechanical arm against the eyelid, and consequently may be uncomfortable to wear and use.
In addition, some devices may detect eye movement, but may not be able to distinguish when the eye is opened or closed. For example, it may be desirable to measure the percentage of total time that the eyelids are closed as a function of time or the area of the palpebral fissure that is covered by the eyelid as the eye is opened or closed, commonly known as xe2x80x9cPERCLOS,xe2x80x9d for example during medical research or when monitoring driver alertness. Devices that merely detect eye muscle movement or eyelash movement may not be able to distinguish when the eye is open or closed, and consequently may not be able to measure PERCLOS. Similarly, such devices may not measure other parameters, such as velocity of eyelid closing or opening, acceleration or deceleration characteristics, duration of open or closed eye states, intervals between eye blinks and/or partial versus full eye blinks or eye closures.
Further, infrared cameras or other devices may be used to monitor a driver""s awareness, which are typically mounted on the dashboard, roof or other fixed mounting within the user""s vehicle. Such devices, however, require that the user maintain constant eye contact with the camera. In addition, they do not monitor eyelid movement if the user looks sideways or downwards, turns around, exits the vehicle or compartment in which he or she is being monitored, or if the camera moves relative to the individual. Further, such cameras may have problems seeing through eyeglasses, sunglasses, or even contact lenses, and may not operate effectively in sunlight.
Accordingly, it is believed that a more effective system and method for monitoring eye and/or eyelid movement would be considered useful.
The present invention is directed to systems and methods for monitoring eye movement. Generally, humans blink at least about 5-30 times per minute, or about 7,000-43,000 times per day. Each involuntary-reflexive blink lasts about 200-300 milliseconds, and generally averaging about 250 milliseconds, amounting to about 1,750-10,800 seconds per day of eye closure due to involuntary blinking. As tiredness or sleepiness occurs, the eye blink gets longer and slower until the eyes begin to close for short term xe2x80x9cmicrosleeps,xe2x80x9d i.e., sleep conditions that last for about 3-5 seconds or longer, or for prolonged sleep. The present invention provides systems and methods for monitoring, measuring, and/or responding to eye movement, e.g., nonpurposeful reflexive eyeblinks.
In a preferred embodiment, the system includes an emitter and a sensor in a predetermined relationship with an eye such that the emitter emits light and the sensor detects light from the emitter, the sensor producing a light intensity signal indicating when the eye is open or closed. More preferably, the emitter is directed or aimed at the eyelid and eye, while the sensor detects eyelid-reflected light, since, unlike the eyelid, the eye ball (except the retina, which may cause a xe2x80x9cred reflexxe2x80x9d under white light conditions or xe2x80x9cwhite pupilxe2x80x9d under infrared light) does not reflect substantial light back to the sensor. Circuitry is coupled to the sensor for converting sequential light intensity signals corresponding to eyelid movement received from the sensor into a stream of data, and a processor converts the stream of data into an understandable message.
The circuitry for converting sequential light intensity signals may compare the sequential light intensity signals with a predetermined time threshold to detect voluntary-intentional or unintentional-involuntary sequences of eyelid movements, corresponding, for example, to a predetermined binary code. Memory circuitry may be coupled to the processor for storing the stream of data and/or a communication device, such as a video monitor or synthesized voice module, may be coupled to the processor for communicating the understandable message. In addition, a control system may be coupled to the processor, and the understandable message may include a command for controlling equipment, including electrical or electronic equipment, machinery, or a computer or computer accessory devices coupled to the control system.
The system preferably also includes a transmitter, preferably a radio frequency transmitter, for wireless transmission of the stream of data to a remote location. Alternatively, other forms of wireless transmission, e.g. infrared, as well as hard-wire connections may be used. The processor, as well as the memory circuitry, communication device, and/or control system, may be located at the remote location, and a receiver may be coupled to the processor for receiving the stream of data from the transmitter.
In a preferred form, the system includes a detection device having a frame adapted to be worn on a person""s head, e.g., with the frame resting on the bridge of the user""s nose and/or ears. The frame has the emitter and sensor thereon such that the emitter and sensor are oriented towards the person""s eye when the frame is worn on the person""s head. Preferably, the emitter and sensor are a single solid state device, such as a biosensor device, that emits light within a predetermined frequency range, for example infrared light, towards the eye and detects the emitted light reflected off of the eyelid, respectively.
In another preferred embodiment, a system for monitoring a blinking cycle of a person from a remote location is provided that includes an emitter for directing light towards an eye, and a sensor in a predetermined relationship with the emitter for detecting the emitted light reflected off of the eye, the sensor producing an output signal indicating when the eye is open or closed. Depending upon the relative position of the emitter and sensor with respect to the moving eyelid, the emitter light may be reflected off of the eyelid back to the sensor, or diffused by the surface of the eyeball.
A transmitter is coupled to the sensor for wireless transmission of the output signal, and a processor is provided for comparing the output signal to a predetermined threshold to detect when the eyelid is closed for a minimum predetermined duration. A warning indicator may be coupled to the processor, the warning indicator being activated when the processor detects that the eyelid is closed for the minimum predetermined duration. For example, the warning indicator may be an audible buzzer, a visible warning light, a vibrating device, an electrical shock device, a gustatory smell device, or other device that may act as a stimulus to any sensory modality.
Similar to the previous embodiment, a receiver may be provided at the remote location coupled to the processor for receiving the wireless transmission from the transmitter. Memory circuitry may be provided for storing the output signal and/or a processor may be provided for converting the output signal into an understandable message. A communication device may be coupled to the processor for communicating the understandable message.
In another preferred embodiment, a self-contained device for detecting movement of a person""s eyelid is provided that includes a frame adapted to be worn on the person""s head, an emitter on the frame for directing light towards an eye of the person when the frame is worn, and a sensor on the frame for detecting light from the emitter. The sensor produces an output signal indicating when the eye is open or closed, and a transmitter on the frame is coupled to the sensor for wireless transmission of the output signal to a remote location. The frame may also include a processor for comparing the output signal to a predetermined threshold to detect drowsiness-induced eyelid movement. Similar to the previous embodiments, the emitter and sensor are preferably a solid state biosensor device for emitting and detecting infrared light, or alternatively an array of emitters and/or sensors in a predetermined configuration on the frame, e.g., in a vertical, horizontal, diagonal, or other linear or other geometric array of more than one emitter and/or sensor oriented towards one or both eyes. In particular, an array of emitters and/or sensors allows measurement of eyelid velocity, acceleration and deceleration, and calculation of xe2x80x9cPERCLOS.xe2x80x9d
The emitter and/or sensors may be affixed to any number of points on the frame, e.g., around the lens and preferably in the nose bridge, or alternatively anywhere along the frame, including near or on the nasal portion of the frame, the attachment of a temple piece of the frame, and/or surface mounted on the lens of an eyeglass. Alternatively, the emitter and/or sensor may be embedded in the lens of an eyeglass, or otherwise such that they operate through the lens. Thus, the emitter(s) and/or sensor(s) are fixed on an eye-frame such that they move with the wearer""s head movements, and continuously focus on the user""s eyes, whether the user is in a vehicle, outdoors or in any other environment.
Thus, a system in accordance with the present invention may detect eyelid movement of the user, distinguish normal blinks from other voluntary or involuntary eyelid movement, and produce a stream of data. The stream of data may be converted into an understandable message, such as a binary code, a command for controlling a piece of equipment, or an indicator of the user""s physical, mental or emotional state. Thus, the system may provide a convenient and/or effective method for voluntary or involuntary communication based simply upon movement of the user""s eye.
In accordance with another aspect of the present invention, a system is provided for monitoring movement of a person""s eye. The system includes a device configured to be worn on a person""s head and an array of emitters on the device for directing light towards an eye of the person when the device is worn. The array of emitters is configured for projecting a reference frame towards the eye. A camera is oriented towards the eye for monitoring movement of the eye relative to the reference frame. The camera may be provided on the device or may be provided remote from the device, but in relatively close proximity to the user.
Preferably, the array of emitters includes a plurality of emitters disposed in a substantially vertical arrangement on the device, and a plurality of emitters disposed in a substantially horizontal arrangement on the device. Thus, the array of emitters may project a focused set of crossed bands towards the eye for dividing a region including the eye into four quadrants.
In addition, the system preferably includes one or more scanning or nonscanning sensors on the device for detecting light from the array of emitters. The one or more sensors produce an output signal indicating when the eye is open or closed, similar to the embodiments described above. More preferably, the sensors include an array of focused sensors in a predetermined relationship with the array of focused emitters for detecting light from the array of emitters that is reflected off of respective portions of the eye or its eyelid. The emitters, because of their fixed position, produce a fixed reflection off of the surface of the eye and eyelid, appearing as a xe2x80x9cglint,xe2x80x9d i.e., a spot or band of light. Each sensor produces an output signal indicating when the respective portion of the eye is covered or not covered by the eyelid.
The system may also include a processor for correlating the output signal from the one or more sensors with a video signal from the camera for determining the person""s level of alertness. The system may also include a warning indicator on the device, the warning indicator being activated when the processor determines a predetermined level of drowsiness has occurred.
Light from the array of emitters may be emitted towards the eye of a user wearing the device to project a reference frame onto the eye. The camera is capable of imaging light produced by the emitters, e.g., in the infrared light range, thereby detecting the projected light as a spot of light, band of light or other xe2x80x9cglint.xe2x80x9d Movement of the eye relative to the reference frame may be monitored with the camera. A graphical output of the movement monitored by the camera relative to the reference frame may be monitored. For example, infrared light from the emitters may be reflected off of the retina as a xe2x80x9cred reflexxe2x80x9d under white light, as a xe2x80x9cwhite pupilxe2x80x9d under infrared light, or as a dark pupil under subtraction, using methods known to those skilled in the art. The processor, using these methods, may detect movement of the eye""s pupil may be measured relative to the reference frame. This movement may be graphically displayed, showing the movement of the eye""s pupil relative to the reference frame.
In addition, the output signal from the one or more sensors may be correlated with video signals produced by the camera monitoring movement of the eye relative to the reference frame, thereby determining the person""s level of drowsiness.
Other objects and features of the present invention will become apparent from consideration of the following description taken in conjunction with the accompanying drawings.